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FINAL  REPORT 
William  L.  Bewley 

CRESST/University  of  California,  Los  Angeles 


Abstract 

The  UCLA  Center  for  Research  on  Evaluation,  Standards,  and  Student  Testing 
(UCLA/CRESST)  and  CRESST's  subcontractors,  University  of  Southern  California 
Behavioral  Technology  Laboratory  (USC/BTL),  the  University  of  Southern 
California  Rossier  School  of  Education  (USC/RSOE)  and  CHI  Systems,  Inc.,  have 
successfully  completed  the  tasks  identified  in  the  Statement  of  Work  distilled  from 
the  additions  and  deletions  due  to  the  six  grants  awarded  over  the  four  years  of  the 
Knowledge,  Models,  and  Tools  to  Improve  the  Effectiveness  of  Naval  Distance 
Learning  (KMT)  project  and  associated  deliverables. 

This  Final  Report  summarizes  six  grants,  the  tasks,  and  deliverables.  The 
deliverables  are  provided  as  attachments  to  this  report. 
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Introduction 

The  Office  of  Naval  Research  (ONR)  Capable  Manpower  Future  Naval 
Capability  (FNC)  program  is  dedicated  to  exploiting  technologies  that  will  help 
Sailors  and  Marines  be  fully  prepared  to  fight  and  win  in  an  information-rich, 
distributed  battlespace.  The  Knowledge,  Models,  and  Tools  to  Improve  the 
Effectiveness  of  Naval  Distance  Learning  (KMT)  project  addresses  one  of  the  three 
Enabling  Capabilities  identified  by  the  Capable  Manpower  FNC:  "Equip  Sailors  and 
Marines  with  effective  mission-essential  competencies  when  and  where  needed  at 
an  affordable  cost."  It  responds  to  all  the  objectives  specified  for  the  Advanced 
Distance  and  Distributed  Learning  Supporting  Technology  identified  the  Capable 
Manpower  FNC  as  required  to  achieve  this  Enabling  Capability.  These  objectives 
are: 

•  Develop  traditional  courseware  with  learner  support  tools  optimized  for 
Advanced  Distributed  Learning  (ADL)  delivery,  using  ADL  coursework 
authoring  capabilities  and  pedagogically  sound  principles. 

•  Develop  guidelines  and  demonstrate  effectiveness  of  ADL  techniques  in 
operational  settings. 

•  Develop  and  demonstrate  advanced  learner  support  tools  including 
collaborative  and  team  training  methods  for  ADL. 

Grants,  Tasks,  and  Deliverables 

Work  on  the  KMT  project  was  supported  by  six  grants  awarded  over  four 
years.  Each  grant  either  redefined  tasks  and  deliverables,  deleting  or  elaborating  on 
some  and  adding  others,  making  it  difficult  for  the  reader  to  track  what  was  to  be 
done.  To  clarify  KMT  tasks  and  deliverables,  we  provide  this  historical  narrative 
summarizing  the  six  grants  and  their  effects  on  tasks  and  deliverables  and  conclude 
with  a  consolidation  of  these  effects  into  a  final  list  of  tasks  and  deliverables. 

1.  The  Initial  KMT  Grant 

To  support  these  objectives,  CRESST  received  a  grant  in  December  2001  to 
perform  the  following  tasks  over  a  three-year  period  of  performance.  The  grant 
funded  the  first  of  the  three  years  of  the  KMT  project. 
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1.  Conduct  Ethnographic  Studies 

2.  Develop  Distance  Learning  Assessment  Models 

3.  Develop  Learning  Environment  Infrastructure 

4.  Prototype  and  Develop  Distance  Learning  Guidelines 

5.  Test  the  Distance  Learning  Guidelines 

6.  Develop  Learner  Support  Strategies 

7.  Transition  Results  to  the  Operational  Environment 

8.  Program  Management 

CRESST  was  responsible  for  Tasks  2,  7,  and  8.  Task  1  was  performed  by  a 
subcontractor,  CHI  Systems,  Inc.  A  second  subcontractor,  the  University  of  Southern 
California  Behavioral  Technology  Laboratory  (USC/BTL),1  performed  Task  3.  A 
third  contractor,  the  University  of  Southern  California  Rossier  School  of  Education 
(USC/RSOE),  was  responsible  for  Tasks  4,  5,  and  6. 

2.  Amendment  to  the  Initial  KMT  Grant 

In  December  2002  an  amendment  to  the  initial  grant  was  funded  for  a  three- 
month  effort  to  (a)  complete  our  FY02  activities  in  the  human  performance 
assessment  and  authoring  areas,  (b)  refine  the  design  of  the  Human  Performance 
Knowledge  Mapper,  (c)  improve  the  validity  studies  associated  with  its  use  in 
Marine  Marksmanship,  and  (d)  submit  a  set  of  revised  guidelines  created  to  support 
instructional  and  assessment  authoring,  with  documentation  of  the  guidelines 
development  methodology.  Activities  (a),  (b),  and  (c)  are  extensions  of  Task  2  above. 
Activity  (d)  was  an  extension  of  Task  4. 

3.  The  KMT  Grant  for  the  Second  Year 

In  December  2002  CRESST  received  a  grant  for  the  second  of  three  years  of  the 
KMT  award.  This  award  redefined  the  tasks  to  be  performed.  The  revised  task  list  is 
summarized  below: 

1.  Develop  Version  2.0  of  the  Human  Performance  Knowledge  Mapper.  Enhanced 
authoring  flexibility  for  measurement  of  declarative  and  procedural 
knowledge;  enhanced  flexibility  for  scoring,  setting  standards,  and 


1  USC/BTL  was  renamed  the  USC  Center  for  Cognitive  Technology  (USC/CCT)  in  2005.  To  avoid 
confusion,  the  name  USC/BTL  will  be  used  throughout  this  final  report. 
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reporting;  trials  in  one  or  two  Navy  task(s);  documentation;  scientific 
validity  study(ies);  and  reports. 

2.  Develop  Version  1.0  of  the  Human  Performance  Problem  Solving  Mapper. 
Program  functionality  to  represent  critical  relationships  and  decisions  in 
Human  Performance  Problem  Solving;  trial  in  embedded  Navy  task(s); 
scientific  study  of  technical  quality  and  validity;  adaptation  of  authoring 
system;  documentation;  design  version  2.0;  and  reports. 

3.  Develop  Version  1.0  of  the  Complex  Human  Performance  Problem  Solving  System. 
Finalize  design;  implement  in  software  for  Web  administration,  scoring  and 
reporting;  apply  to  one  or  two  Navy  task(s);  conduct  pilot  study  and  study 
of  technical  quality;  documentation;  design  version  2.0;  and  reports. 

4.  Develop  Assessment  Decision  Support  Tools.  Prepare  technical  and  policy 
papers  on  psychometric  and  statistical  issues  in  implementing  the  Five- 
Vector  Model;  conduct  data  analyses  using  available  or  simulated  data; 
adapt  CRESST  software  to  be  used  to  aggregate  individual  performance 
reports  to  provide  data  ultimately  needed  for  the  Five-Vector 
implementation;  and  reports. 

5.  Develop  iRides  Performance  Simulation-Instruction  Delivery  System  and 
Authoring  System.  Voice  I/O  capabilities;  advanced  graphics  capabilities; 
improved  student  records /data  foundations;  automatic  collection  of 
performance  records;  extend  core  instruction  features  (action  monitoring); 
efficiencies,  performance  tuning  (speed,  memory  footprint  improvements); 
integrated  simulation  and  instruction  authoring;  simulation  and  instruction 
debugging;  integration  of  COTS  graphics  authoring  and  simulation 
authoring;  instruction  templates  and  template  interfaces;  authoring  with 
instructional  strategies;  documentation;  and  reports. 

6.  Develop  Draft  Human  Performance  Learning  and  Measurement  Guidelines. 
Develop  draft  guidelines  in  Metacognition  and  Motivation;  implement  02 
Guidelines  in  Navy  and  Marine  example(s);  implement  Metacognition  and 
Motivational  guidelines  in  Navy  domains  with  examples;  self-monitoring 
instructional  module  implementation  in  Navy  domain;  conduct  validity 
study;  and  reports. 

7.  Program  Management 

This  redefinition  resulted  in  a  major  revision  of  KMT  tasks.  First,  it  eliminated 
Task  1  and  CHI  Systems,  Inc.,  as  a  subcontractor.  CHI's  first-year  work  is  included 
in  the  interim  report  on  rifle  marksmanship  described  below  and  attached  to  this 
final  report.  Second,  the  original  Task  2,  "Development  of  Distance  Learning 
Assessment  Models,"  was  decomposed  into  three  more  specific  tasks — the  new 
Tasks  1,  2,  and  3  above — in  addition  to  development  and  testing  of  non-mapper 
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assessment  models.  Third,  the  two  original  guidelines  tasks,  Tasks  4  and  5,  were 
combined  into  the  new  Task  6.  Fourth,  the  initial  Task  6,  Develop  Learner  Support 
Strategies,"  was  dropped.  The  initial  Task  3  is  not  affected,  corresponding  to  the 
new  Task  5. 

4.  The  KMT  Grant  for  the  Third  Year 

In  January  2004  CRESST  received  a  grant  for  the  third  of  three  years  of  the 
KMT  award,  to  continue  work  on  the  tasks  described  above. 

5.  An  Investigation  of  the  Reliability  of  Knowledge  Measures 

In  January  2004  CRESST  also  received  funding  added  to  the  KMT  award  to 
conduct  an  investigation  of  the  reliability  of  knowledge  measures  through  relational 
mapping  in  joint  military  environments.  The  purpose  of  this  task  was  to  use  the 
CRESST  knowledge-mapping  system  to  assess  individual  trainee  knowledge  (i.e.,  a 
trainee  maps  his  or  her  understanding  of  the  domain  using  the  online  knowledge 
mapping  tool),  and  then  conduct  generalizability  and  decision  studies  to  examine 
the  psychometric  properties  of  knowledge  mapping  scores  to  evaluate  the  suitability 
of  knowledge  mapping  as  an  assessment  of  trainees7  understanding  of  joint  mission- 
essential  tasks. 

6.  Performance  Assessment  Tools  for  Distance  Learning  and  Simulation 

In  November  2004  CRESST  received  funding  added  to  the  KMT  award  to  build 
performance  assessment  tools  for  distance  learning  and  simulation.  The  goal  was  to 
produce  a  knowledge-map  assessment  authoring  system  that  can  be  used  to 
generate  performance  assessments  integrated  with  an  instructional  simulation 
toolkit  provided  by  a  subcontractor,  the  University  of  Southern  California  Center  for 
Cognitive  Technology  (CCT).  The  authoring  system  will  support  Navy  and 
contractor  instructional  design/development  personnel  who  are  not  assessment 
experts  to  develop  performance-based  assessments,  including  specification  of 
scoring,  consistent  with  modern  practices  and  focused  on  assessing  deep 
understanding  of  a  domain  such  as  equipment  maintenance.  The  assessments  can  be 
embedded  in  instruction  delivered  through  a  simulation-based  training  system. 
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The  Consolidated  Task  List  and  Deliverables 

Table  1  summarizes  the  final,  consolidated  list  of  KMT  tasks  and  deliverables 
distilled  from  the  additions  and  deletions  due  to  the  six  grants  awarded  over  the 
four  years  of  the  project.  For  each  task,  the  table  identifies  the  grant  in  which  it 
became  a  requirement,  the  performer  responsible  for  its  completion,  and  the 
associated  deliverable,  designated  by  a  letter,  A  through  F.  Deliverables  are 
identified  in  Table  2.  The  letter  designations  correspond  to  those  used  in  Table  1  to 
associate  tasks  to  deliverables. 


Table  1 

The  Consolidated  List  of  KMT  Tasks  and  Deliverables 


Task 

Grant 

Performer 

Deliverable 

1.  Conduct  Ethnographic  Studies 
(Year  1  only) 

1 

CHI  Systems,  Inc. 

A  • 

2.  Development  of  Distance  Learning 
Assessment  Models 

1 

UCLA/CRESST 

A,  B,  C 

3.  Develop  Version  2.0  of  the  Human 
Performance  Knowledge  Mapper 

2 

UCLA/CRESST 

D,E  # 

4.  Develop  Version  1.0  of  the  Human 
Performance  Problem  Solving  Mapper 

2 

UCLA/CRESST 

D,  E 

5.  Develop  Version  1.0  of  the  Complex  Human 
Performance  Problem  Solving  System 

2 

UCLA/CRESST 

F 

• 

6.  Assessment  Decision  Support  Tools 

2 

UCLA/CRESST 

G 

7.  Develop  iRides  Performance  Simulation- 
Instruction  Delivery  System  and  Authoring 
System 

2 

USC/BTL 

H 

• 

8.  Develop  Draft  Human  Performance  Learning 
and  Measurement  Guidelines 

2 

USC/RSOE 

I 

9.  Transition  Results  to  the  Operational 
Environment 

1 

UCLA/CRESST 

J 

10.  Investigate  the  Reliability  of  Knowledge 
Measures 

5 

UCLA/CRESST 

K  # 

11.  Performance  Assessment  Tools  for  Distance 
Learning  and  Simulation 

6 

UCLA/CRESST 

L 

12.  Program  Management 

All 

UCLA/CRESST 

• 
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Table  2 

KMT  Deliverables 

Designation  Title 

A  Interim  Report:  Research  on  USMC  Marksmanship  Training  Assessment  Tools, 

Instructional  Simulations,  and  Qualitative  Field-Based  Research 

B  Determinants  of  Rifle  Marksmanship  Performance:  Predicting  Shooting  Performance 

with  Advanced  Distributed  Learning  Assessments 

C  New  Tools  and  Methods  for  Assessing  Risk-Management  Strategies 

D  CRESST  Human  Performance  Knowledge  Mapping  System 

E  CRESST  Human  Performance  Knowledge  Mapping  Tool  Authoring  System 

F  An  Architecture  for  a  Problem-Solving  Assessment  Authoring  and  Delivery  System 

G  Psychometric  Issues  Related  to  the  Five- Vector  Model 

H  iRides  Performance  Simulation  /  Instruction  Delivery  and  Authoring  Systems 

I  What  Works  in  Distance  Learning:  Guidelines 

J  KMT  Transitions 

K  An  Investigation  of  the  Reliability  of  Knowledge  Measures  Through  Relational 

Mapping  in  Joint  Military  Environments 

L  Performance  Assessment  Tools  for  Distance  Learning  and  Simulation 


Deliverables 

Each  deliverable  is  described  in  the  following  sections  and  is  attached  to  this 
Final  Report.  The  exception  is  KMT  Transitions,  which  is  only  described  in  this 
report. 


A: 

Interim  Report:  Research  on  USMC  Marksmanship  Training  Assessment  Tools, 
Instructional  Simulations,  and  Qualitative  Field-Based  Research 

The  Office  of  Naval  Research  (ONR)  has  funded  the  UCLA  National  Center  for 
Research  on  Evaluation,  Standards,  and  Student  Testing  (CRESST)  and  its 
subcontractor,  the  University  of  Southern  California  Behavioral  Technology 
Laboratories  (BTL),  to  conduct  research  on  the  integration  of  assessment  models  and 
tools  with  instructional  simulation  authoring  and  delivery  tools  designed  to  support 
Navy  and  Marine  Corps  distance  learning  (DL).  An  assessment  model  is  a  formal 
specification  of  tasks  and  measures  producing  data  that  can  be  used  as  evidence  to 
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support  inferences  about  what  the  learner  knows.  For  this  project  the  assessment 
models  and  tools  provide  precision  measurement,  analysis,  and  diagnosis  of  learner 
performance  guiding  selection  and  delivery  of  remedial  training  through  the 
instructional  simulations.  ONR  is  also  funding  CHI  Systems,  Inc.,  to  conduct  field- 
based  qualitative  research  on  Navy  and  Marine  Corps  DL  implementations  to 
develop  practical  guidelines  and  procedures  to  support  effective  DL  employment  in 
the  Navy  and  Marine  Corps. 

The  initial  focus  of  these  research  efforts  was  USMC  marksmanship  training  at 
the  Camp  Lejeune  Weapons  Training  Battalion.  CHI  Systems,  Inc.,  conducted  a 
qualitative  field-study-based  approach  to  assess  the  effectiveness  of  existing  DL 
while  the  CRESST/BTL  effort  used  the  marksmanship  training  context  to  explore 
new  options  for  DL  assessment  and  training.  Following  intensive  knowledge 
acquisition  to  build  an  appropriate  marksmanship  knowledge  base  to  guide 
assessment  development  and  scoring,  CRESST  created  and  pilot  tested  several  DL- 
delivered  assessment  tools  including  knowledge  mapping,  a  prior  knowledge 
questionnaire,  shot  group  depiction,  a  background  survey,  a  self-regulation  survey, 
and  an  assessment  of  rifle  marksmanship  measuring  Marines'  skill  at  identifying 
proper  and  improper  firing  positions.  Data  are  still  being  analyzed,  but  preliminary 
results  indicate  that  such  cognitive  measures  in  combination  with  background 
information  can  indeed  predict  shooter  performance  and  do  it  as  well  as  or  better 
than  far  more  expensive  and  time-consuming  measures.  CRESST's  data  recently 
collected  at  Camp  Lejeune's  Stone  Bay  Rifle  Range  indicate  that  a  combination  of 
background  variables  and  cognitive  measures  predict  qualification  scores  with  a 
multiple  regression  coefficient  of  .59.  Benchmarks  for  comparison  are  the  best  results 
obtained  using  performance  on  a  rifle  simulator  (Hagman,  1998),  which  show  a 
correlation  of  .69,  and  a  Marine's  most  recent  qualification  score,  which  shows  a 
correlation  of  .40. 

To  enable  use  of  assessment  results  in  selecting  and  delivering  remedial 
training,  BTL  and  CRESST  have  developed  and  prototyped  a  way  to  integrate 
assessment  results  and  interactive  distributed  training.  Simulations  and  simulation- 
centered  training  modules  for  marksmanship  training  were  developed  using  BTL's 
iRides  authoring  and  delivery  systems.  iRides  and  iRides  Author,  originally 
developed  with  ONR  support,  have  continued  to  be  enhanced  and  extended  in  the 
current  ONR-sponsored  project.  Simulation-based  training  modules  have  been 
developed  for  the  Battlesight  Zero  procedure  and  shot  group  analysis.  Additional 
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modules  on  creating  the  proper  sight  picture  and  data  book  usage  are  under 
development. 

CHI's  field-based,  qualitative  study  of  Marine  Corps  marksmanship  training 
employed  extensive  interviews  with  trainers,  trainees,  and  administrators  to  provide 
an  overall  assessment  of  training  quality.  CHI  found  widespread  dissatisfaction  with 
current  Phase  I  training,  because  of  the  inconsistency  with  which  it  appears  to  be 
conducted.  Existing  advanced  technology  was  found  to  be  little  used.  Extensive 
"down  time"  for  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  since  its 
introduction  has  led  trainers  and  trainees  to  have  little  confidence  in  its  availability 
during  the  brief  Phase  I  training.  Problems  (whether  real  or  perceived)  of  skill 
transfer  from  the  simulator  to  the  real  firing  range  also  deterred  ISMT  usage.  In 
addition,  limited  number  of  available  computers  constrained  use  of  the  digitized 
marksmanship  ADL  course.  The  course  was  also  not  integrated  into  the  Phase  I 
curriculum  in  a  clear  way,  leading  trainers  away  from  its  use. 

The  combined  efforts  of  CHI,  CRESST,  and  BTL  provide  an  overall  picture  of 
the  current  state  of  DL  in  USMC  marksmanship  training  and  prototype  assessment 
and  instructional  simulation  tools  of  potential  benefit  to  USMC  marksmanship 
training  through  early  identification  and  remediation  of  Marines  likely  to  shoot 
poorly.  Early  identification  and  remediation  could  lead  to  savings  in  travel  cost,  time 
away  from  the  Marine's  home  unit,  the  coaches'  time  on  the  firing  line,  the  use  of 
firing  range  capacity,  and  the  cost  of  ammunition  and  targets.  In  addition  to  savings, 
early  identification  and  remediation  could  lead  to  higher  scores  overall,  fewer 
UNQs,  and  more  Experts.  To  realize  these  potential  benefits,  the  following  work 
should  be  completed.  The  first  two  tasks  would  require  transition  funding. 

•  Validation  of  the  assessments'  ability  to  go  beyond  prediction  of  overall 
shooting  performance  to  identify  specific  knowledge  gaps,  predict  the 
impact  of  the  missing  knowledge  on  shooting  performance,  and  identify  the 
remediation  required  to  fill  the  gap. 

•  Shrink-wrapping,  bullet-proofing,  and  validating  a  complete  assessment 
and  training  package  based  on  the  prototypes. 

•  Qualitative  studies  of  the  results  of  fielding  the  assessment  and  training 
package  produced  with  transition  funding,  including  analyses  of  how 
organizational  processes  adapt  (or  need  to  adapt)  to  make  effective  use  of 
DL  delivery,  and  guidelines  and  a  supporting  case  study  helping  the  Navy 
(via  NETC)  and  Marine  Corps  (via  the  DLC)  better  design  and  employ  DL. 
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In  addition,  the  following  work  might  be  considered  for  additional  R&D 
funding: 


•  Integrating  assessment  and  training  with  the  forthcoming  Marine  LOMAH 
(Location  Of  Miss  And  Hit)  system.  This  may  make  it  possible  to  assess 
shooting  performance  directly  and  automatically,  and  to  select  appropriate 
training  modules  based  on  that  performance. 

•  Hand-held  assessment  and  training  modules.  These  could  be  used  to  teach 
data  book  activities  in  the  context  of  real  shooting  environments,  with 
automatic  evaluation  and  feedback.  Advice  could  be  provided  on  the  firing 
line,  and  instructors  could  be  automatically  alerted  about  problems.  If 
integrated  with  the  LOMAH  system,  these  hand-held  devices  could  also 
provide  feedback  and  advice  about  actual  shooting  performance. 

The  deliverable  is  attached  to  this  Final  Report. 

B: 

Determinants  of  Rifle  Marksmanship  Performance:  Predicting  Shooting 
Performance  with  Advanced  Distributed  Learning  Assessments 

The  UCLA  National  Center  for  Research  on  Evaluation,  Standards,  and  Student 
Testing  (CRESST)  is  under  contract  to  the  Office  of  Naval  Research  (ONR)  to 
conduct  research  on  assessment  models  and  tools  designed  to  support  Navy  and 
Marine  Corps  distance  learning  (DL).  The  first  such  application  is  in  support  of 
USMC  marksmanship  training.  In  a  series  of  studies  we  examined  the  role  of 
cognitive  and  non-cognitive  variables  in  the  prediction  of  rifle  marksmanship 
performance.  Prior  research  on  predicting  shooting  performance  suggests  a 
deceptively  complex  task  sensitive  to  a  variety  of  variables.  The  stages-of-skill- 
development  model  (Ackerman,  1987,  1992;  Fitts  &  Posner,  1967)  suggests  cognitive 
measures  will  be  most  sensitive  to  individuals  in  the  learning  phase,  and  perceptual- 
motor  measures  most  sensitive  to  individuals  past  the  learning  phase.  The  role  of 
cognitive  variables  (knowledge  of  shooting  in  particular)  is  largely  unexplored 
beyond  examination  of  shooting  performance  across  groups  receiving  different 
training  and  instruction. 

In  a  series  of  studies  we  were  able  to  predict  record-fire  performance  between 
.52  to  .86,  depending  on  the  sample.  Bivariate  correlations  between  various 
measures  and  record-fire  scores  were  obtained  in  the  .2  to  .8  range.  Perceptual- 
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motor  measures — intended  to  reflect  experience — were  consistently  a  good 
predictor  of  performance.  The  most  recent  record-fire  score  predicted  record-fire 
score  at  the  .3  to  .4  range.  The  best  single  predictor  of  record-fire  score  was  the  firing 
line  experience  survey,  which  yielded  correlation  coefficients  from  .6  to  .8.  Cognitive 
measures  (aptitude  and  knowledge  related  to  marksmanship)  in  less  experienced 
samples  related  to  record-fire  score  in  the  .2  to  .4  range.  No  relationships  between 
record-fire  score  and  knowledge  measures  were  found  in  the  more  experienced 
sample.  Affective  measures  (worry,  anxiety)  predicted  record-fire  scores  in  the  -.3  to 
-.6  range  and  in  general,  for  the  affective  and  firing  line  experience  measures,  state 
measures  had  coefficients  of  higher  magnitude  than  the  trait  versions. 

Overall,  we  have  gathered  evidence  that  in  general  suggests  a  knowledge 
component  to  shooting  performance.  The  results  of  our  studies  point  to  differences 
in  knowledge  of  rifle  marksmanship  between  participants'  pre-classroom  training 
and  post-classroom  training,  between  more  experienced  participants  and  less 
experienced  participants,  between  high  performers  and  low  performers,  and 
between  higher  aptitude  and  lower  aptitude  participants.  Knowledge  measures  can 
predict  record-fire  scores  moderately  in  less  experienced  samples,  and  when 
combined  with  other  variables  within  the  stages-of-skill-processing  framework,  can 
predict  record-fire  scores  as  well  as  scores  from  a  rifle  simulator. 

Rifle  marksmanship  is  a  complex  psychomotor  skill  sensitive  to  variations  in 
the  individual,  equipment,  and  environment.  It  is  unlikely  that  variation  in  the 
equipment  and  environment  can  be  reduced  much,  thus  leaving  the  individual  as 
the  only  area  for  improvement.  Given  that  we  have  found  a  cognitive  component  to 
rifle  marksmanship  performance,  it  may  be  that  improving  a  Marine's  knowledge  of 
rifle  marksmanship  will  have  the  most  cost-effective  payoff.  Early  identification  and 
remediation  could  lead  to  increased  cost  savings  in  travel,  decreased  time  away 
from  the  Marine's  home  unit,  increased  throughput  on  the  firing  line,  increased  time 
coaches  spend  providing  feedback  to  shooters  on  the  firing  line,  and  lower 
ammunition  and  target  costs.  In  addition  to  cost  savings,  early  identification  and 
remediation  could  lead  to  higher  scores  overall  and  fewer  unqualified  Marines. 

The  deliverable  is  attached  to  this  Final  Report. 
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C: 

New  Tools  and  Methods  for  Assessing  Risk-Management  Strategies 

At  the  request  of  the  Office  of  Naval  Research  (ONR),  we  provided  the  U.S. 
Navy  two  tools  to  help  evaluate  the  process  that  novice  acquisition  officers  use  to 
integrate  risk -management  strategies  with  the  federal  military  acquisition  process. 
The  Human  Performance  Knowledge  Mapping  Tool  (HPKMT)  was  designed  to 
evaluate  the  ability  of  subjects  to  depict  the  relationships  among  the  key  phases  in 
the  acquisition  process  and  subject  understanding  of  risk  management.  The  Decision 
Analysis  Tool  (DAT)  allowed  subjects  to  use  Expected  Value  and  Multi-attribute 
Utility  Theories  to  evaluate  the  risks  and  benefits  of  various  acquisition  alternatives, 
and  allowed  us  to  monitor  the  process  subjects  used  to  arrive  at  a  procurement 
decision.  When  we  evaluated  the  HPKMT  knowledge  maps  of  17  subjects  against 
expert  maps  developed  by  their  instructors,  we  found  that  subject  understanding  of 
incorporating  risk  management  in  the  acquisition  process  trended  higher,  but  did 
not  improve  significantly.  Sequential  analysis  of  data  from  the  DAT  allowed  us  to 
isolate  distinct  risk-management  strategies,  as  well  as  strategies  that  overly  focused 
on  (or  ignored)  aspects  of  risk  management.  The  use  of  a  referent  to  determine  the 
conceptual  relationships  and  strategic  acquisition  skills  necessary  to  be  a  skilled 
acquisition  officer  are  discussed  as  extensions  of  this  work. 

The  deliverable  is  attached  to  this  Final  Report. 

D: 

CRESST  Human  Performance  Knowledge  Mapping  System 

CRESST  designed  and  developed  a  knowledge  mapping  tool  intended  to 
measure  content  understanding.  The  deliverable  presents  a  review  of  knowledge 
mapping  scoring  methods  and  current  online  mapping  systems,  and  the  overall 
design,  functionality,  scoring,  usability  testing,  and  authoring  capabilities  of  the 
system.  While  there  exist  several  tools  available  to  construct  knowledge  maps, 
CRESST's  knowledge  mapping  tool  is  one  of  the  only  systems  designed  specifically 
for  assessment  purposes,  the  only  system  that  can  support  multiple  assessment 
formats,  and  the  only  system  with  an  empirical  base.  Limitations  of  the  system  and 
possible  next  steps  are  discussed. 

The  deliverable  is  attached  to  this  Final  Report. 
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E: 

CRESST  Human  Performance  Knowledge  Mapping  Tool  Authoring  System 

Effective  delivery  of  advanced  distributed  learning  (ADL)  training  to 
individuals  in  a  Naval  environment  requires  tools  to  support  the  creation  and 
administration  of  assessment  tasks.  We  have  developed  a  knowledge  mapping 
authoring  system  intended  to  be  simple  and  user-friendly  in  its  interface,  but  highly 
functional,  requiring  a  minimal  number  of  clicks  to  navigate.  The  authoring  system 
is  intended  to  allow  a  diverse  set  of  users  to  create,  modify,  adapt,  and  reuse 
knowledge  mapping  tasks.  Various  scoring  options  provide  information  on  student 
performance  on  different  dimensions  and  at  different  levels  of  stringency.  The 
system  stores  student  data  and  thus  performance  can  be  monitored  over  time.  This 
report  introduces  knowledge  mapping  and  provides  guidelines  on  the  creation  of 
knowledge  mapping  tasks,  and  then  describes  the  operation  of  the  CRESST  Human 
Performance  Knowledge  Mapping  Tool  Authoring  System. 

The  deliverable  is  attached  to  this  Final  Report. 

F: 

Architecture  for  a  Problem-Solving  Assessment  Authoring  and  Delivery  System 

This  report  describes  the  design  of  an  authoring  system  to  support  the  design 
of  problem-solving  assessments.  A  key  component  underlying  the  system 
architecture  is  a  constraint  network.  In  a  constraint  network,  nodes  are  variables  that 
can  assume  a  range  of  values  and  the  topology  specifies  how  the  variables  and 
values  are  related  (Montanari,  1974).  To  support  assessment  design,  the  system 
design  includes  a  constraint  network  describing  the  permissible  relations  and  states 
among  assessment  and  problem-solving  variables. 

The  deliverable  is  attached  to  this  Final  Report. 

G: 

Psychometric  Issues  Related  to  the  Five-Vector  Model 

The  Navy's  five-vector  model  provides  a  comprehensive  specification  of  the 
requirements  for  performance-based  advancement  of  military  personnel.  The 
effectiveness  of  the  model  for  making  performance-based  advancement  decisions 
will  depend  heavily  on  the  psychometric  quality  of  the  individual  measures  and  on 
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the  psychometric  models  that  are  used  to  combine  the  information  from  the 
measures  that  comprise  each  of  the  vectors  of  the  model.  The  psychometric 
challenges  related  to  the  five  vectors  have  some  things  in  common,  but  each  of  them 
also  has  some  unique  considerations.  We  begin  by  considering  the  vectors  one  at  a 
time,  and  then  discuss  challenges  in  combining  information  across  the  five  vectors  to 
make  performance-based  advancement  decisions.  The  results  of  our  analysis  suggest 
that  it  is  likely  that  the  measurement  information  will  vary  across  and  within  vectors 
in  terms  of  validity  and  reliability,  that  the  profile  of  information  is  likely  to  be 
uneven  across  vectors,  and  the  information  from  one  vector  may  even  conflict  with 
that  from  another  vector.  These  properties  can  be  expected  to  make  the  tasks  of 
combining  and  integrating  the  information  across  vectors  quite  challenging. 

The  deliverable  is  attached  to  this  Final  Report. 

H: 

The  iRides  Performance  Simulation  /  Instruction  Delivery  and  Authoring  Systems 

USC/BTL  developed  iRides,  an  advanced  system  for  delivering  authored 
interactive  graphical  simulations  and  instructional  vignettes.  The  system  provides 
the  ability  to  deliver  simulation-based  instruction  in  three  ways:  as  Java 
applications,  as  Java  applets,  and  as  Java  Web  Start  applications.  The  latter  two 
options  make  it  possible  for  these  authored  interactive  graphical  simulations  and 
training  to  be  delivered  over  the  Web  or  any  similar  network,  thereby  supporting 
advanced  distributed  learning. 

A  new  authoring  tool,  iRides  Author,  was  developed  for  developing 
simulations  and  instruction  that  iRides  can  deliver.  This  tool  is  a  Java  application 
that  provides  drawing  tools,  behavior  authoring  and  behavior  debugging  interfaces, 
and  instruction  authoring  and  instruction  debugging  interfaces.  The  instruction 
approach  supports  the  development  of  novel  instructional  "routines"  that  can  be 
authored  in  XML  and  reused  in  multiple  contexts.  The  iRides  instruction  authoring 
system  can  be  easily  extended  to  provide  simple  user  interfaces  that  support  the 
development  of  instruction  using  such  instructional  templates. 

The  applet  version  of  iRides  can  be  delivered  as  a  SCORM-compliant  shareable 
content  object  (SCO).  This  feature  is  currently  being  exercised  in  a  Marine  Corps 
transition  project,  in  which  an  iRides  SCO  collaborates  with  the  MarineNet  Learning 
Management  System  (LMS). 
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An  additional  tool  for  authoring  iRides  simulations  and  training  was  also 
produced.  This  tool,  Rivets,  is  a  fast  C++  program  that  has  been  compiled  for  three 
different  Unix-type  operating  systems:  Linux,  Silicon  Graphics  IRIX,  and  Macintosh 
OS  10.3  or  later  with  XI 1.  It  is  now  possible  to  author  iRides  simulations  and 
training  on  a  variety  of  platforms,  including  all  the  Windows  platforms  from 
Windows  98  to  the  present. 

The  flexible  and  open  architecture  of  iRides  makes  it  possible  to  employ  this 
tool  in  collaboration  with  other  advanced  training  system  components,  such  as 
intelligent  tutors.  Additional  work  remains  to  be  done  to  make  iRides  a  plug-in 
component  that  can  be  used  with  a  variety  of  advanced  tutoring  system  approaches. 

The  deliverable  is  attached  to  this  Final  Report. 

I: 

What  Works  in  Distance  Learning:  Guidelines 

Guidelines  were  produced  by  using  experts  to  compile  research-based 
knowledge  using  a  standardized  format.  The  following  list  summarizes  guideline 
categories  and  experts  who  developed  them: 

1 .  Multimedia  Strategies,  by  Richard  Mayer 

2.  Instructional  Strategies,  by  Richard  Clark 

3.  Learning  Strategies,  by  Myron  Dembo  and  Linda  Gubler  Junge 

4.  Assessment  Strategies,  by  Eva  Baker,  Zenaida  Aguirre-Munoz,  Jia 
Wang,  and  David  Niemi 

5.  Motivation  Strategies,  by  Richard  Clark 

6.  Self-Regulation  Strategies,  by  Harold  F.  O'Neil  and  Sanhui  (Sabrina) 
Chuang 

7.  Management  Strategies,  by  Edward  Kazlauskas 

A  paper  version  was  created  and  transitioned  to  the  training  and  education 
community  as  a  published  book.  The  deliverable,  the  published  book  referenced 
below,  is  attached  to  this  Final  Report. 

O'Neil,  H.  F.  (2005).  What  works  in  distance  learning:  Guidelines.  Greenwich,  CT: 

Information  Age  Publishing,  Inc. 
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J: 

KMT  Transitions 

Work  performed  on  the  KMT  project  has  led  to  two  transitions:  (1) 
marksmanship  assessment  tools  transitioned  to  the  USMC  Weapons  Training 
Battalion  with  $701,000  in  6.4/6. 5  funding  from  the  USMC  College  of  Continuing 
Education;  and  (2)  a  risk  management  decision  tool  transitioned  to  the  USN 
Engineering  Duty  Officer  (EDO)  School  with  no  additional  funding. 

The  USMC  Rifle  Marksmanship  Coaches  Course  Toolset 

The  USMC  Weapons  Training  Battalion  seeks  to  enhance  rifle  marksmanship 
proficiency  at  reduced  cost  by  use  of  assessment  tools  measuring  knowledge  of 
fundamentals  of  rifle  marksmanship.  Four  products,  collectively  called  the  Rifle 
Marksmanship  Coaches  Course  Toolset,  have  been  transitioned. 

The  Evaluation  of  Shooting  Positions  module  is  a  Web-based  tool  that  allows 
the  user  to  manipulate  a  virtual  shooter  on  a  computer  display  to  judge  whether  the 
shooter's  position  is  proper  or  improper.  If  the  user  judges  any  element  to  be 
improper,  the  Marine  can  adjust  the  shooter  to  the  proper  position.  Scoring  is 
automated  to  enable  cost-effective  online  use  in  distance  learning. 

The  Data  Book  Training  module  provides  Web-based  interactive  training  on 
the  use  of  the  data  book,  used  by  every  Marine  to  record  actual  shooting 
performance  and  consequent  adjustments  of  rifle  sights  based  upon  wind 
conditions,  distance  to  target,  and  firing  speed.  The  module  also  provides  interactive 
training  on  sight  picture,  battle  sight  zero  (BZO)  and  shot  group  analysis.  Learner 
performance  is  scored  automatically. 

The  Marksmanship  Knowledge  Mapper  assesses  deep  understanding  of 
marksmanship  knowledge — concepts  and  procedures  and  how  they're  related,  with 
automated  scoring  to  make  it  cost-effective.  The  Fundamentals  of  Marksmanship 
task  delivered  by  the  Mapper  assesses  knowledge  of  the  cause-effect  relations 
among  marksmanship  fundamentals.  A  separate  Shot-to-Shot  Analysis  tool  was 
initially  developed  as  a  mapping  task  but  was  transitioned  into  a  standalone  Web- 
based  module.  The  Shot-to-Shot  Analysis  tool  assessed  the  learner's  knowledge  of 
the  relation  of  shot  patterns  to  shooter  problems,  e.g.,  breath  control. 
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The  Marksmanship  Inventory  Knowledge  Assessment  module  is  a  Web- 
based  survey  measuring  a  student's  knowledge  of  marksmanship  fundamentals  that 
can  be  quickly  administered  with  automated  scoring  prior  to  or  following 
instruction. 

Funding  for  the  USMC  transition  was  provided  by  the  USMC  College  of 
Continuing  Education  (CCE):  $531,722  in  FY04  and  $168,582  in  FY05,  for  a  total  of 
$700,304. 

The  USN  EDO  School  Decision  Analysis  Tool 

USC/BTL's  iRides  system  was  used  to  build  an  application  for  modeling 
decisions  where  it  is  possible  to  estimate  the  probability  and  utility  of  the  possible 
outcomes  of  a  series  of  such  decisions.  This  application  can  be  used  to  practice  this 
approach  to  complex  decisionmaking.  It  can  also  provide  instruction  about  its  own 
usage  and  about  case  studies  depicted  in  its  interface.  The  EDO  School  faculty  has 
incorporated  this  tool  into  portions  of  the  EDO  basic  class. 


Deliverables  A  and  B  describe  the  KMT-funded  research  leading  to  the  USMC 
transition.  Deliverable  C  describes  the  research  leading  to  the  USN  EDO  School 
transition. 


K: 

An  Investigation  of  the  Reliability  of  Knowledge  Measures  Through  Relational 
Mapping  in  Joint  Military  Environments 

The  goal  of  this  task  was  to  gather  evidence  on  the  effectiveness  of  online 
knowledge  mapping  as  a  method  to  assess  high-level  understanding  of  specific 
military  domains  and  tasks.  We  used  the  CRESST  Human  Performance  Knowledge 
Mapping  Tool  (HPKMT)  to  assess  individual  trainee  knowledge  and  then  examined 
the  psychometric  properties  of  knowledge  mapping  scores  to  evaluate  the  suitability 
of  knowledge  mapping  as  an  assessment  of  trainees'  understanding  of  joint  mission- 
essential  tasks. 

Students  attending  the  Joint  Special  Operations  University  in  Hurlburt,  Florida, 
created  three  knowledge  maps  for  three  content  areas:  Air  Tasking  Order  (ATO) 
cycle.  Joint  Task  Force  Structure  and  Function  (JTF),  and  Joint  Special  Operation 
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Task  Force  Structure  (JSOTF).  Because  an  insufficient  number  of  participants  was 
provided,  we  were  unable  to  complete  the  planned  generalizability  analysis,  but 
analyses  of  scoring  techniques  yielded  important  information  about  the  quality  of 
the  knowledge  maps,  and  the  assessments  provided  valuable  information  regarding 
student  understanding  of  JSOU  course  content. 

The  student  maps  were  analyzed  using  three  methods:  automated  criterion- 
based  (expert)  assessment,  propositional  analysis,  and  structural  mapping  analysis. 
The  criterion-based  assessment  showed  significantly  lower  scores  for  the  students  as 
compared  to  experts  for  both  tasks.  The  propositional  analysis  found  that  the  expert 
and  student  use  of  terms  and  links  were  fairly  proportional,  with  some  exceptions. 
The  structural  analysis  revealed  differences  between  expert  maps  and  student  maps, 
and  differences  among  students'  maps  relative  to  structural  complexity.  In  general, 
the  expert  maps  had  more  terms;  variable  use  of  source,  sinks,  and  carriers; 
numerous  clusters;  and  high  reachability.  Additionally,  a  comparison  of  a  sample  of 
student  maps  revealed  similar  patterns,  with  more  sophisticated  maps  containing  a 
higher  number  of  terms,  links,  and  clusters  as  well  as  level  of  reachability.  In 
addition  to  these  research  results,  we  were  able  to  create  a  standalone  version  of  the 
mapper  that  has  been  used  in  subsequent  studies  for  the  military. 

The  deliverable  is  attached  to  this  Final  Report. 

L: 

Performance  Assessment  Tools  for  Distance  Learning  and  Simulation 

Performance  assessment  tools  for  distance  learning  and  simulation  were 
developed  by  extending  the  UCLA  Center  for  Research  on  Evaluation,  Standards, 
and  Student  Testing  (UCLA/CRESST)  knowledge  mapping  tool's  authoring  and 
scoring  functionality  and  providing  the  capability  to  embed  a  knowledge  mapping 
assessment  in  simulation-based  training  developed  by  the  University  of  Southern 
California  Center  for  Cognitive  Technology  (USC/CCT).  Products  are  described  in 
detail  including  the  knowledge  mapping  tool  with  authoring  and  scoring  systems, 
performance-based  assessments,  an  instructional  simulation  providing  interactive 
training,  and  the  performance-based  assessments  embedded  in  the  instructional 
simulation. 

The  deliverable  is  attached  to  this  Final  Report. 
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